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Abstract 

We consider the problem of modelling noisy but 
highly symmetric shapes that can be viewed as 
hierarchies of whole-part relationships in which 
higher level objects are composed of transformed 
collections of lower level objects. To this end, 
we propose the stochastic wreath process, a fully 
generative probabilistic model of drawings. Fol¬ 
lowing Leyton’s ’’Generative Theory of Shape”, 
we represent shapes as sequences of transforma¬ 
tion groups composed through a wreath product. 

This representation emphasizes the maximiza¬ 
tion of transfer — the idea that the most compact 
and meaningful representation of a given shape 
is achieved by maximizing the re-use of existing 
building blocks or parts. 

The proposed stochastic wreath process extends 
Leyton’s theory by defining a probability distri¬ 
bution over geometric shapes in terms of noise 
processes that are aligned with the generative 
group structure of the shape. We propose an in¬ 
ference scheme for recovering the generative his¬ 
tory of given images in terms of the wreath pro¬ 
cess using reversible jump Markov chain Monte 
Carlo methods and Approximate Bayesian Com¬ 
putation. In the context of sketching we demon¬ 
strate the feasibility and limitations of this ap¬ 
proach on model-generated and real data. 


1 INTRODUCTION 

A substantial part of human knowledge and reasoning is 
based on the idea of a part-whole hierarchy in which an 
entity at a given level represents a part of a whole at a 
higher level and is itself composed of parts defined at a 
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lower level. This notion is crucial to the natural sciences 
(elementary particle, atom, molecule, cell, organ, organ¬ 
ism, etc), but is also at work in the structure of documents 
(letter, word, sentence, paragraph, section, chapter, book), 
music (single note, chord, chord progression, piece etc.), 
or architecture (wall, room, wing, building). Even plans of 
action are often best represented in terms of hierarchies of 
goals and subgoals. While it may be debatable if reality 
itself can be said to exhibit this kind of structure, it is clear 
that the human mind frequently resorts to the principle of 
hierarchy in organizing complex structure. In this paper, 
we take a mathematical formulation of this idea provided 
by Michael Leyton (SJ based on the group-theoretic notion 
of wreath product and show how one can build probabilis¬ 
tic models of shape that discover a hierarchical generative 
representation - providing the basis for understanding and 
manipulating the shape. 

The view of computer vision as inverse computer graphics 
is very elegant and has a long history in the field. The key 
idea is to define a graphics language to describe the gener¬ 
ative process for creating a class of images and—given an 
image—to infer its generative history in terms of that lan¬ 
guage. To account for irregularities, noise, and ambiguity, 
we define a stochastic rendering process that mitigates the 
rift between the platonic graphics language and the reality 
of the image m .While it is difficult to make this paradigm 
work for general classes of images, we focus on the special 
case of hand-drawn, highly symmetric geometric sketches. 

What would be a good graphics language to describe ge¬ 
ometric sketches? Typical drawing tools (including those 
in PowerPoint) provide graphics primitives such as points, 
lines, circles and squares. Using grouping, copy, and align¬ 
ment it is possible to create figures that re-use certain el¬ 
ements of the figure to ensure consistency and regularity. 
However, the true underlying constraints and regularities 
are often lost because the full generative history is not rep¬ 
resented. As a consequence, it is often difficult to edit a 
sketch while preserving its underlying structure. 

In his book ”A Generative Theory of Shape” O, Michael 
Leyton proposes a graphics language that is totally genera- 





tive and captures what he calls the maximisation of transfer 
and recoverability. The key idea is to describe the emer¬ 
gence of shape as a generative process that unfolds struc¬ 
ture from previously unfolded substructures — eventually 
going back to a single point: the origin. The maximisa¬ 
tion of transfer means that as far as possible the shape is 
’’explained” by re-using existing building blocks. Once a 
given shape is understood in terms of such a totally gener¬ 
ative history, it can be intelligently manipulated by chang¬ 
ing sub-structures (which may appear repeatedly in the un¬ 
folded shape), completing an incomplete shape based on 
the inferred regularities or using it as a building block in a 
super-structure. 

To make Leyton’s theory practical, we introduce the 
stochastic wreath process, which generalizes Leyton’s for¬ 
malism to the case of noisy shapes. While Leyton’s gen¬ 
erative theory of shape characterises a given highly regu¬ 
lar shape, the stochastic wreath process represents a dis¬ 
tribution over shapes—which have irregular appearance 
but highly regular structure. The noise process factorizes 
across the different hierarchical levels of the shape (one per 
group factor in the chain of wreath products), and hence is 
perfectly aligned with the generative process. 

The stochastic wreath process allows us to make Leyton’s 
theory practical in the sense that for a given hand-drawn 
sketch of a shape we can infer a posterior distribution over 
generative histories in terms of the wreath process. To ex¬ 
plore this idea, we define a rendering pipeline based on 
the wreath process which generates actual pixel images 
and propose a reversible jump MCMC method |4) inspired 
by Approximate Bayesian Computation fiTH for inference. 
Note that the model class we describe can also be viewed 
as a domain specific probabilistic programming language 
with the inference process attempting to synthesize appro¬ 
priate models. 


2 A GENERATIVE MODEL OF SHAPE 

2.1 LEYTON’S GENERATIVE THEORY 

Leyton (8) characterizes the structure of a shape by the (or¬ 
dered) sequence of actions on the canvas that led to its cre¬ 
ation, its generative history. In a broad sense, geometric ob¬ 
jects are seen as memory stores of a set of actions. These 
actions are modelled by a series of (algebraic) groups of 
transformations. 


2.1.1 Preliminaries 

Definition (Groups) A group (G, o) is a nonempty set G 
together with a binary operation o on G that satisfies the 


following properties: 

V£i,02 G G,gi og 2 G G 
gi ° (92 °93) = (gi °g 2 ) °£ 3 , 
V01,02,03 G G 

Be G G, s. 1 e o g = g o e = g,Vg G G 
\/g G G, Bh G G, s.t. goh = hog = e 

and h will be denoted by g~ x . 

Definition (Group action) Let G be a group and A be a 
set. G is said to act on A if there is a map a such that: 
a(e, x) = x,Vx e X where e is the neutral element of G 
and that a(g, a(h, x)) = a(g o h , x), Vg, h G G. 

Commonly the abbreviation a(g,x) = gx is used. 

In this paper we consider three major families of transfor¬ 
mation groups. These transformations act on a two dimen¬ 
sional vector space M 2 . To simplify the presentation, we 
work within an extended space with elements in M 3 , where 
the third dimension is set to unity such that affine transfor¬ 
mations can be expressed as matrix multiplications. 

• Translations (along the X and Y axis): 

G T = {T t \tel}^{A Tt \tel} (1) 
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for translations along the A-axis. And which can be 
either continuous (X = M) or discrete (X = Z). 


• Rotations (about the origin O) of discretization n: 


G R = {R k \keX}^{A Rk \keX} (2) 
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and which can be infinite (X = Z n with n —>> oo), or 
discrete (X = Z n , with n G N, n > 2, finite), where 
Z n denotes the cyclic group of order n. 


• Mirroring/Reflexion (along the A and Y axis): 


G M = {M k \kel}^{A Mk \kel} (3) 
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for mirroring along the A-axis, and k G X = Z 2 . 








2.1.2 Generating/Drawing a line 

Let us consider the scenario of drawing a horizontal line: 
we start with a point, call it p , and in order to create a line, 
we translate this point along the horizontal direction. We 
will model this action by a continuous translation group 
as defined in Eq. [T] More precisely, for each element in 
translation-group T t we make a copy of the point p , call it 
p t , and then transfer p t to the desired location by letting T t 
act on it: this will produce a new point on the canvas at t 
distance away from p. 

If we were to apply the whole (continuous) translation 
group Gt{X), where X = M, we obtain an infinite line 
around the starting point p. This is not always desirable, in¬ 
stead more commonly we would like to draw a (bounded) 
line-segment. This can be done by noticing that in order 
to create a segment we need only a subset of the elements 
of the full translation group considered above. We will call 
the subset of indices of these group elements (that are pre¬ 
sented in the picture) the occupancy set, occ C X, asso¬ 
ciated with the desired shape. In the case of continuous 
groups, we allow occ to be specified as an interval (i.e. the 
unit segment centred around the origin will have occupancy 
[—0.5, 0.5]) and for the discrete cases, occ can be an arbi¬ 
trary selection of the available indices in X. Furthermore, 
we will often use the following terminology to describe the 
occupancy set: full occupancy if occ = X\ single occu¬ 
pancy if | occ| = 1 (set contains only 1 element) or arbi¬ 
trary otherwise. 



Figure 1: Wreath Product: Generation of a square 


translation transformation that maps the origin onto p — it 
will leave no trace. After this initial translation of the ori¬ 
gin, the generative process continues as described above. 

2.1.4 Transfer as a Wreath product 

In both scenarios what happens amounts to two steps: we 
make copies of the original shape, one for each element in 
the transformation group and then for each of these copies 
we apply the corresponding group element to transfer it to 
its desired location, form, or orientation. This construction 
is algebraically modelled by the wreath product between 
the initial shape generative history and the transformation 
group we want to employ in the transfer. 


2.1.3 Generating/Drawing a square 

Let us move on to more complex structures and consider 
describing the generative process behind drawing a square. 
Remember that the central idea of this generative process 
is maximization of transfer. We have previously seen how 
to characterize the generative process of drawing a line- 
segment, which could be one side of the square. Let us start 
by drawing the top side of the square. To create a square we 
wish to transfer this side via a 4-fold rotation group (Eq. [2]). 
As before, we start by creating copies of the top side, one 
for each element in the transformation group for a total of 
four. Then we let each element of the group act on its copy 
to create each side of the square. The process is depicted in 
Figure [T] 

The generative process of the square, as described so far, 
starts with any point p on the top side or — given the right 
choice of occupancy — indeed anywhere on the implied 
infinite line of which the side is a finite segment. In this 
paper, all the generative processes we are considering will 
start at the origin (defined as the centre point of the canvas). 
Thus, the complete generative history of a square will start 
by translating the origin to a point p on the top side of the 
square. This translation will have an occupancy set of car¬ 
dinality 1, containing only the index corresponding to the 


Definition (Semidirect products) Consider H and N two 
groups, with their respective group operations o H and o N 
and a group homomorphism r : H —» Aut[N] Q Now, let 
G be the set of ordered pairs (h, n), with h G H, n G N. 
We can define a binary operation o G on G as follows: 

(ni, hi) o G (n 2 , h 2 ) = {r(h 2 )[n i] o N n 2 , hi o H h 2 ) 

for all (ni, hi), (n 2 , h 2 ) G G. Then under this operation, 
G is a group, denoted by N(s) r H and referred to as the 
semidirect product of N and H under r. 

Definition (Wreath product) Let A and H be two groups 
and let Cl be a set with H acting on it. Let A n be the direct 
product of the copies of A Q := A indexed by the set Cl: 
A Q = Then we can define the action of H on 

A Q in a natural way by letting the group action of H act on 
the indices of the product: 

ELefi ELefi A hu> 

(^CJl > 2 1 * * * ) ^ (®huj 1 1 1 ' ‘ * ) 

1 Aut[N] is the automorphism group of N. An automorphism 
of an object is a isomorphic map from the object to itself - i.e a 
mapping from the object to itself preserves the structure. Addi¬ 
tionally, it can be shown that the set of all automorphisms of an 
object forms a group under the composition operation. 






















Given this action, the wreath product of A and H is defined 
as the semiproduct [A n ]® r H, where r is implicitly given 
by the action above, r : h \—> flh . 

We can see from this definition that in order to define a 
wreath product we need two groups and a set. The first 
one, A, will correspond to the generative history of the 
initial shape So that we want to transfer and the second 
one, H , will correspond to the transformation group used 
in the transfer]^] And the set ft corresponds to the set of 
indices 1 associated with the transformation group. At this 
point, it should be clear that the descriptions of the gener¬ 
ative structures will be groups under full occupancy. Par¬ 
tial occupancy, i.e., the situation in which only a subset of 
elements of the shape is displayed, can be integrated into 
the group-theoretical framework by appending cyclic Z 2 
switches to the fibre copies before application of the con¬ 
trol group. These can be seen as colour or on/off switches 
for parts of the shape. 

2.1.5 Shapes as n-fold wreath products 

We would like to apply the same concept of transfer several 
times in the formation of an image in order to maximize re¬ 
usability. For instance consider that once we have a square 
we would like to form a circle of four such squares. Using 
the ideas highlighted above, we could transfer the already 
constructed square by another 4-fold rotation group. But 
first, recall that the rotations we consider are only around 
the origin and the previously formed square is centered 
about the origin - thus if we were to apply a 4-fold rotation 
to this square, we will end up with four coinciding copies of 
this square. Therefore, first we need to translate the square 
by the intended radius of the circle we want to form. This 
will give rise to a new shape that can be characterized by a 
5-fold wreath product: 

G 0 ®Gi®G2®G3@G4®G 5 

where 

• Go = {e} is the trivial group corresponding to the 
origin (it transfers the origin onto itself) 

• Gi is the continuous translation group G\ = M, re¬ 
sponsible for the vertical translation of the origin to 
the point p on the top side. 

occi = 0.5 C M(= XgJ 

• G 2 is the continuous translation group G 2 = M, re¬ 
sponsible for producing an infinite line. 

occ 2 = [-0.5, 0.5] C M(= Tq 2 ) 

2 Terminology used in Leyton’s literature: the generative his¬ 
tory of the initial shape So that we would like to transfer is re¬ 
ferred to as the fibre-group and will be denoted by G(F ); the 
group responsible for the transfer of So’s generative history will 
be referred to as the control-group and will be denoted by G(C). 


• Go is the first 4-fold rotation group G3 = Z4, respon¬ 
sible for producing a square. 

0CC3 = {0,1, 2, 3 } Z 4 (= Xg 3 ) 

• G 4 is the discrete translation group G 4 = Z, respon¬ 
sible for translating the formed square 2 units away 
from the origin (anticipating the rotation) 

0 CC 4 = {2} G Z(= Tq a ) 

• G 5 is the second 4-fold rotation group G 5 = Z 4 , re¬ 
sponsible for producing the circle of squares. 

OCC 5 = {0,1,2,3} = Z 4 (= 1g 5 ) 

2.1.6 A Grammar for Shapes 

While Leyton develops his theory as abstract mathematics, 
our concern is to have a concrete representation suitable for 
probabilistic inference. Hence, we introduce the following 
grammar to represent shapes. 

S ::= [(Gi, occi);... ; (G n , occ n )] 

G ::= Trans axis | Rot m | Mirror 

axis ::= X | Y 

occ ::= [k 1 ;.,.-k n ]\[r 1 ,r 2 ] 

A shape S denotes (1) the generative history, the n-fold 
wreath product Go®Gi® • • • ®G n where Go = {e} is 
the implicit trivial group corresponding to the origin, but 
also ( 2 ) the associated occupancies occi for each level, that 
characterize the elements of this structure observed in the 
picture. 

We write our circle of squares example as follows: 

[(Trans Y, [0.5, 0.5]), 

(Trans X, [—0.5, 0.5]), (Rot 4, [0..3]), 

(Trans X, [2]), (Rot 4, [0..3])] 

2.2 THE WREATH PROCESS: STOCHASTIC 
SHAPE 



Figure 2: Noise model. For compactness, we omit the sin¬ 
gle occupancy levels in the generative history: but there are 
part of the structure and will take noise instances as well 













To model hand-drawn sketches, we define a noise pro¬ 
cess that accounts for the imperfections present in free¬ 
hand drawings, and that arises naturally by perturbing the 
generative history of the intended shape. More precisely, 
each transformation present in the generative history of our 
shape will have a noise level e which accounts for the er¬ 
ror made by the user when trying to perform the transfer 
corresponding to that transformation. 

Let us consider that the intended structure (shape) is given 
by an n-fold wreath product (Go@Gi@ • • • ®G n ) then 
\/g E Gi,Vi, a level of noise will be sampled to account 
for each application of each element in the control-group. 
In the generative history an element of a group Gi is ap¬ 
plied multiple times, as many times as it is copied by the 
levels above it. Although in the exact shape the copies of 
these transformations are the same, under the noise pro¬ 
cess each of these copies receives its own perturbation in¬ 
dependent of the noise instances corresponding to other 
noisy copies: \/g^ E G[^ sample e^ ~ pc^e), where 
j E occi+i x • • • occ n and can thought of as the coordinate 
of the subtree on which gG) we act, out of the copies of 
that fibre-group that were created by repeat transfer (each 
time a new transformation group was applied). Defining 
the probability distribution pg{c) of possible perturbations 
for a group G we assume that there exits an embedding of 
G into a bigger (continuous) group, on which both noisy 
and non-noisy transformations are defined. ^ 


Table 1: Samples of the wreath process for the square, four¬ 
fold rotation of the square 
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For us, this is trivially the case as we have already consid¬ 
ered this embedding when defining the continuous occu¬ 
pancy. Both the continuous and discrete versions of the 
translation and rotation groups can be embedded in the 
continuous groups. The noisy transformations obtained 
by composing g with the sampled noise instances are ap¬ 
plied to the corresponding fibre-copies. We denote the set 
formed by these noise actions on the 2D plane as Gi = 
{g^ E Gi} for each control group Gi. The process 
is illustrated in Figure [2j Under this interpretation we can 
define a noisy shape as S ~ VF(-|<S). In the following we 
will denote by J\f the set of all noise instances correspond- 

3 Usually there is a natural way of defining this embedding, but 
we could also construct an embedding by another wreath product, 
if there is no other more trivial embedding. 


ing to a shape S. 

2.3 PRIOR OVER HAND-DRAWN SHAPES 

Under the Bayesian paradigm, we need to specify a prior 
over the model’s parameters (<S, AT): 

Prior over generative histories p(S) 

p(S) = p([(Gi,occi )-,...; (Gn,occ n )}) 

n 

= Y[p(ocCi\Gi)p(Gi) ( 4 ) 

i= 1 

where, in the our particular case of transformation groups: 
p(Gi) is considered to be uniform at any level i. The prior 
over occi reflects a strong preference for single (with prob¬ 
ability p) or full occupancy (with probability p) and with 
(1 — 2 p) probability we will consider special occupancy, 
which will be sampled as follows: if \X\ < oc then a par¬ 
ticular element of the index set is to be switched on with 
probability otherwise we define a bounding parame¬ 
ter B (sampled around the origin) which restricts \1\ to a 
finite set and then we will sample uniformly within that re¬ 
striction. For continuous occupancy, in this work, we will 
restrict ourselves to line-segments of unit length and full 
circles. 

Noise instances prior p(J\f\S): 

n 

p(U\S) = n ^ i\0M,S) 

i=l 

n 

= n II p(eF|0JV,Gi) (5) 

1 = 1 jEoCCi X ••• XOCC n 

where Ojg- are the hyper-parameters governing the distribu¬ 
tion of the noise instances and ~et\ is the vector of noise 
instances at level i. This vector has | occi | x • • • x |occ n | el¬ 
ements. The prior over the noise instances is as follows: 
for translation e trans ~ Af(0, cr trans ) where cr trans ~ 

r(l, 20 ); for rotation: p(e Rot ) = ( G^ ) with 

°Rot ~ r( 7 r/n, n 2 ), where Io{k) is the Bessel function of 
order 0 (Io{k) = ^ e kcosx dx ); for mirroring: we do 
not consider an explicit noise action, but by the way we de¬ 
fined the wreath process, a mirrored copy of a given noisy 
shape So will be different from So, as this noisy copy of 5o 
will have its own noise instances—sampled independently 
of the noise instances of So- 

3 INFERENCE 

Consider the grey-value mapping Id of an image. Our aim 
is infer the generative history of the shape in this image. 
This amounts to inferring a structure S representing the n- 
fold wreath product describing the underlying symmetries 





























present in the shape, plus their observed occupancy. Using 
Bayes’ rule, we can express the posterior probability as: 

p(S\I d )kp(I d \S)p(S) (6) 

Thus, for this computation, we need to specify and evaluate 
a prior over generative histories of shapes and a likelihood 
that evaluate the input data on a given shape S. The prior 
encodes our beliefs about the kind of shapes we expect to 
see and may be chosen conveniently to ensure tractabil- 
ity. That leaves us with the computation of the likelihood, 
which in this case is non-trivial as S and Id occupy very 
different domains. However, this fairly common problem 
that can be addressed using approximate Bayesian compu¬ 
tation (ABC) methods that bypass the (exact) evaluation of 
the true likelihood. The idea is as follows: given a parame¬ 
ter setting 0 , a dataset D can be simulated for the stochastic 
model specified by 6. Then a distance measure, p, can be 
defined between the input data D and D (that now live in 
the same space). If the simulated data does not match the 
input data within a given tolerance, i.e. p(D , D ) > e, then 
the set of parameters will be rejected. This idea was par¬ 
ticularized to MCMC simulations in iflOll and lT2l . Note 
that these require the specification of the threshold e, which 
might require further inference. However, more recently 
in fl4l and IE it was shown that the hard specification 
of this threshold could be replaced by a stochastic likeli¬ 
hood model. Our inference will combine these ideas, as 
described in the next section. 

3.1 ABC FOR THE GENERATIVE HISTORY OF 
SHAPES 

We start by rendering the image corresponding to S to get 
into the same domain as the data Id using a deterministic 
rendering function f(S ) = Is . In principle, we could de¬ 
fine a measure on the space of images and compute how 
far the exact rendering of our proposed model is with re¬ 
spect to the input image Id • But defining such a measure is 
problematic, as most such measures will induce very sharp 
distributions on the rendered image Ir. The likelihood will 
yield high values for exact and almost exact matches, while 
most other models will be given a likelihood close to zero. 
In other words, this approach will likely fail to discriminate 
close solutions from arbitrary proposals, except for the un¬ 
likely case that an (almost) perfect match is rendered. 

To overcome this problem, we make use of the ideas in 
approximate Bayesian computation highlighted before and 
follow the approach in (9). We estimate the likelihood 
P(Id\S) using a stochastic likelihood model based on a 
stochastic image renderer. Instead of rendering the exact 
image, we render a noisy version of it as illustrated in Ta¬ 
bles [T] In addition, to increase stochasticity we apply a 
Gaussian blur to the rendering, specified by X = {w^, a^}. 

S^Id^Ir (7) 


where Id is generated via the wreath process, given S and 
Ir is the image resulting from applying a Gaussian blur 
of a 5 and a window size of 2Under this formulation 
and keeping in mind that the rendering function although 
stochastic by nature, becomes deterministic given J\f and 
Y, the posterior probability can be approximated as: 

p(S\I D ) cc [p(S) p(Ir\S,M, X) p(Af, X\S)p(I d \Ir) 

oc f p(S)S fi sM,x) (Ir)p(J^\S)p(X)p(I d \Ir) 

Prior over the parameters of the Gaussian blur p(X): ~ 

b w • Beta( 1, 2 ) and <75 ~ b a • T( 1 , 1 ). 

3.2 EMPIRICAL LIKELIHOOD 

The stochastic render will produce a noisy instance of 
model Ir against which the input image Id can be eval¬ 
uated. To this end, we define the empirical likelihood, as¬ 
suming a Bernoulli distribution in pixel space: 

p(Id\Ir) = n IrM I d[x ' v \ 1 - Ir[x,v]) { 1 ~ Id[x ’ v]) 

[x,y] 

where Ir[x, y] G [0,1] denotes the (grey-scale) intensity of 
pixel located at [x, y] in image Ir and is interpreted here as 
the probability of that pixel being black. 

3.3 REVERSIBLE JUMP-MCMC FOR THE 
WREATH PROCESS 

In this section, we will give a general-purpose algorithm for 
inference in wreath processes, based on Reversible Jump- 
MCMC (introduced in (4|, and refined in 0), but partic¬ 
ularize it in the proposals to exploit the structure of the 
wreath product. The idea is the following: we assume that 
the upper-level structure (of the top level groups) has the 
greatest impact on the appearance of a shape and hence 
should be kept more stable than lower-level parts of the 
generative history. In other words, lower levels can be ex¬ 
plored given upper levels, but changes in upper levels are 
likely going to lead to major revisions in the lower levels. 
Assuming that the higher-level structure has been detected, 
we propose objects on which this upper structure acts by 
transfer. Let us look at what Algorithm[T] amounts to in this 
case. Given a model 0 n = {S,N,X) where n indicates 
that S is an n-fold wreath product, we wish to propose a 
new model 0 n > = (5', AT', X r ). First with a given proba¬ 
bility we choose which parameters to resample: <S, J\f or X. 
Varying S , in most cases varies Al too. The other two types 
of parameters can be sampled independently and when pos¬ 
sible we would like to keep all other parameters fixed—we 
are making local changes only in one dimension type at a 



Algorithm 1 Reversible-Jump Markov Chain Monte Carlo 
(RJ.O) Start with random model (ko : 0 ko ), of non-zero 
probability. 

loop 

(RJ.l) Propose a visit to model m k > with probability 

j(k -> k 1 ) 

(RJ.2) Sample u ~ q(u\Q k , k , k')(proposal density) 

(RJ.3) Let the new sample be defined by (0 k ',u f ) = 
9(k,k') (6k, u), where g( k ,k') A bijective with its inverse 
being g( k ',k) an d random variables u and v! play the 
role of matching the dimensionality of the embeddings 
(0 k ',u f ) and (0 k ,u): i.e. dim(O k /) + dim(u') = 
dim(O k ) + dim(u ) 

(RJ.4) Accept new model (k',6 k /) with probability 
a{9k -> Ok') = min (l ,Ao k ^e k ,), where: 


ie k ^e k , 


end loop 


7i iOk',k') j(k' -)■ k) q(u'\O k >,k',k) 


n{O k ,k) j(k - 

dg(k,k')(0k,u) 


{Ok,u) 


k') q(u\O k ,k,k') 

( 8 ) 


time. Thus in our case: 
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corresponds to keeping the higher-level symmetries and 
changing the object on which these symmetries act. The 
nature of these proposals keeps a nested structure over 
models, that with sampling components from the prior 
greatly simplifies the computation of the acceptance ratio 
in Eq. [8j as the determinant factor is always 1 and several 
other simplifications are possible as the prior factorizes 
over levels and new fibre groups are sampled from this 
prior. 


Table 2: One sample run, illustrating various types of pro¬ 
posals: (a)—Kb) Change in occupancy of the top level trans¬ 
lation; (b)—Kc) Changing the fibre-group(from a circle to 
a cross), but keeping the top level structure; (c)—Kd) An¬ 
other change in the fibre-group, as the previous rotation did 
not quite fit the data; (d)^(e) Change in the blur param¬ 
eters: once the structure(at least up to occupancy) is in¬ 
ferred; (e)—Kf) Change in the noise instances. 


Input 




K). 


where A is a global scaling factor, that is sampled uniformly 
between [1,50]. This corresponds to our (inferred) unit. 
Freehand sketches might not respect a standardized unit 
(like 1cm), but we postulate that the user has in mind an 
implicit grid with this A as unit interval. 

Noise instances proposal: q(J\f J\f f ) = p(J\f'\S) 


Intermediate iterations 


(a) (b) (c) (d) (e) (f) 


4 EXPERIMENTS 

In the following, we describe a set of experiments, under¬ 
taken to illustrate the usability of the wreath process in dis¬ 
covering symmetry structures in noisy 2D shapes. 


Gaussian blur proposal: q(X —>> X') = p(X') are sam¬ 
pled from the prior, which was previously described. Let 
us concern ourselves with the proposals in the structure of 
the wreath product: 

Shape proposal: q(S S') Given S = 

[(Gi, occi);...; (G n , occ n )], we propose a new structure 
and implicitly a new shape as follows. We have two 
main types of moves: one that changes the dimensionality 
of the structure through n, and one that does not. The 
moves within a model (keeping n constant) will change 
the occupancy sets or the individual groups, but will 
prefer to move within the same family of transformation 
groups. Changes in the dimensionality of the wreath 
product are as follows: we pick a random level i (with 
higher probability for lower i-s), at which to segment the 
structure. We keep the upper-level structure from level 
i + 1 to n and we re-sample (from the prior) the group 
on which this structure acts. As mentioned before, this 


Noise instances M as part of the model: 

Firstly, we constructed a data set of 10 images sampled 
from the previously defined shape prior, which we will re¬ 
gard as ground truth. For each of these, we will consider: 
the exact rendering of the wreath product sampled and a 
hand-drawn version of it. On this dataset we perform two 
type of inferences: one as described in Section [33] and one 
without accounting for the noise described by the Wreath 
process - in this case the sampling procedure is similar to 
the one presented in Section |3.3| but we do not have any 
proposals involving J\f. This initial experiment was done 
to assess the impact and importance of keeping track of the 
noise instances J\f explicitly in the model. As a result of 
this experiment, we observed a substantially higher average 
recoverability rate of structure, especially for hand-drawn 
images. Thus, the rest of the experiments were carried out 
by inferring a model M = (S, A/*), although in most cases 
the quantity we are primary interested in is S. 


















































Measure of performance: 

We have defined two measures to quantify our results: 

• (Full) Recoverability - the inferred structure matches 
the group structure, or an equivalent version, includ¬ 
ing the right occupancy sets. 

• Recoverability up to occupancy - the wreath product 
has been successfully inferred, but the occupancy is 
not quite right. 

An example of correctly inferred structure, but with slightly 
off occupancy is the example in Table |3.3| where the top 
level translation is inferred correctly, and so is the rota¬ 
tion group before it, but the occupancy there accounts for 
three elements being switched on whereas we actually ob¬ 
serve only two. It is important to note that this is by no 
means optimal in assessing recoverability of structure, as 
for instance, the last two examples in Table [3] will score 
0 under both these measures, although clearly a lot of the 
structure, in particular the higher level control groups, are 
recovered. Unfortunately, quantifying partial recoverabil¬ 
ity is very problematic because of the equivalence between 
models under the 2 D projection on the canvas and the lim¬ 
ited occupancy. In general, there are several possible expla¬ 
nations of a partially observed structure, and various ways 
of constructing the same object. 

4.1 RECOVERING SAMPLES FROM THE PRIOR 


Table 3: Sample Runs 
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To evaluate our model and explore its capabilities of re¬ 
covering structure, we construct a data set of 50 examples 
sampled from the prior. We limit the level of complexity 
to compositions of at most 8 groups.Complexities higher 
than that tend to lead to highly dense images, for which 
the number of copies of basic fiber groups tends to be very 
high and require a high resolution to properly distinguish 
them. On the other hand, if the occupancy is low, for such 
a dense structure, there is very little information to differ¬ 
entiate between possible explanations - such cases usually 
look random to the naked eye. Sample runs can be seen in 
Tableland quantitative results are reported in Table 4J_ 


4.2 APPLICATIONS 

Recovering structure from partial occupancy. 

In our dataset, we included partial occupancy at different 
levels and by the nature of our sampling we will re-visit and 
propose occupancy changes with higher probability at the 
top levels. In the context of sketches, these examples cor¬ 
respond to partially observed structure - unfinished draw¬ 
ings - for which, given enough copies, the intended struc¬ 
ture can be inferred and be employed to make suggestion 
or automatic fill-ins. Some examples of such samples can 
be viewed in Table [5] We also report the sample with the 
highest likelihood, I ml, and the sample with the highest 


Table 4: Recovering structure from sample from the prior 


Data Set 

Recoverability 

Recoverability up to occ. 

Exact Shape 

42.76% 

55.84% 

Hand-drawn 

39.65% 

52.28% 


occurrence in the posterior, I map- This is not a simple 
inference problem: usually the full structure is inferred be¬ 
fore a perfect recovery is achieved, thus the sampler has 
to be quite confident in the discovered structure in order to 
overcome not explaining fully the data, which is penalized 
by the likelihood. This trade-off is mediated by the noise 
instances and the blurring intensity. Once the model starts 
to match well parts of the input, the blur width and vari¬ 
ance start to decrease, which increases the likelihood. This 
is only possible if the noise instances A f match well the per¬ 
turbations present in the input. This is why accounting for 
A f in the model was found to be essential for hand-drawn 
samples. 

Common regular structures 

As most of the examples sampled from the prior looked 
rather abstract, expecially the ones with more complex 
structure, we look at some more common regular struc¬ 
tures. First, we look into recovering regular polygons. 
We have already seen the example of the square and 
































































Table 5: Recovering structure from partial occupancy 
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Inferred image 

Inferred Structure 
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[(Trans X), (Rot 27r), 
(Trans X), (Rot 4), 
(Trans Y)] 


we can express any regular polygon in a similar fash¬ 
ion. One way of doing it is: starting with the ori¬ 
gin, we translate it horizontally to make a line, then 
translate the line vertically, and then perform the in¬ 
fold rotation. Below is the description in our grammar: 
(Trans X, [—£,£]), (Trans Y, [ft]), (Rot n, [0 ..n — 1]). 

The above control group, applied to the origin, will pro¬ 
duce an n-sided regular polygon, of length L = 2t and 
height ft. The structure is quite simple and arises naturally 
in various samples from the prior, but exact recovery is a 
more challenging problem as we have a strong preference 
for occupancy that represents integer multiples of the scale 
unit A. In general, we found that this is a reasonable as¬ 
sumption, but in this particular case there is a deterministic 
relation between L and ft and in most of the case there is 
no choice of A that will assure both L and ft to be integer 
multiples of such a unit. The recovery of the rotation and 


Table 6 : Different ft = 1..3, keeping the side length fixed 



[(Trans X, [—3,3]), (Trans Y, [—ft,]), 
(Rot 5, [0..4])] 


translation symmetries is not affected, but the determinis¬ 
tic/constraint relationship between L and ft of exact regular 
polygons actually encodes additional structure. This can be 
captured by a slight change to the control group^j 


Table 7: Regular polygons 
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control group described in Eq. [9] and used it in the pro¬ 
posal mechanism, as preferred structure. We report the 
shape scoring the maximum likelihood and the one with 
the highest posterior. Inference will prefer simpler expla¬ 
nations, as it can be seen from the third and fourth example. 
The complexity of the original shapes is slightly higher, but 
it explains well actual fiber copies in the input, and in fact 
most of the pixels in the input. In principle, we could force 
the likelihood to penalize more unexplained pixels, if full 
recoverability if important. But as seen before a more for¬ 
giving likelihood allows for inference of structure with un¬ 
observed copies. Depending on the application, one would 
need to trade complexity and fidelity to the input, Id • 


Table 8 : Predefined regular structures 



Architectural sketches: Floor Plans 

Lastly, we download floor plans sketches of two famous 
buildings: the Dome of the Rock [^] representative of Is¬ 
lamic architecture and the Villa La Rotonda^\ landmark of 
Palladian architecture, as presented in Table [9] 

5 RELATED WORK 


(Scale Z, [-1]), (Trans X, [-£,£]), (Scale Z, [1]), (9) 

(Trans Y, [ft]), (Rot n, [0..n — 1])] 

where l = (2£) -1 ft tan 7 r/n. The control group above can 
be applied to any other fiber group to create a regular poly¬ 
gon using this fiber group as the building block. 

Predefined Grid-like structures. 

We also tried out some common regular structures (Table 
[ 8 j, like grids, having regular polygons as fiber groups. To 
speed up the inference, we predefined the regular polygon 

4 We define the scaling group Gs( A) = {X k h\k Gl = Z} 


Wreath products have been applied previously to com¬ 
puter vision and image processing, in particular for mul¬ 
tiresolution analysis generalizing approaches based on 
Haar/Fourier transform Eld. 

Treating vision as an inverse inference problem aims to 
estimate the causes and factors that describe a generative 
history - generally proposing some hierarchical representa- 

“http://en.wikipedia.org/wiki/Dome_of_ 
the_Rock 

c http://en.wikipedia.org/wiki/Villa_ 
Rotonda 







































































































Table 9: Floor plans inference results 
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tions. These usually employ a bottom-up generative pro¬ 
cess, coupled with some kind of top-down validation and 
have been successfully used in image and scene parsing, 
but they usually require expert knowledge in setting up the 
hierarchy and encoding a known high level structure(like 
spacial relationships between objects/primitives) El E). 
In contrast, the wreath process automatically detects this 
structure by maximization of transfer. 

Most literature on sketch beautification often employs 
beautification by recognition: they provide a vocabulary of 
primitives and any object in the data must be represented 
in this vocabulary. The approach has limited generaliza¬ 
tion by itself. More recently, the idea of constructing more 
complex objects out of a group of easily detectable primi¬ 
tives was used in tuna. Note that such methods could be 
used in conjunction with the wreath process. 

6 CONCLUSION 

In summary, the three main contributions of this paper 
are: We propose the stochastic wreath process as a new, 
highly structured random point process, thus generalizing 
Leyton’s generative theory of shape; We propose an infer¬ 
ence scheme for inferring structure and parameters of the 
wreath process for a given observed pixel image; We re¬ 
port on experimental results of the inference based on both 
model-generated as well as hand-drawn images of geomet¬ 
ric shapes. 

While our experiments were restricted to the domain of two 
dimensional monochromatic geometric figures, the same 
kind of hierarchical generative model can also be applied 
to three-dimensional shapes. Also, as mentioned in Ley¬ 
ton’s book, the action of the group itself can be different 
from inking and could also include cutting away of mate¬ 
rial or similar shape-creating actions. 

Finally, a wreath product representation can be viewed as 
providing a natural coordinate system for a shape in the 
most general sense. For example, in the case of the square, 
the wreath product representation provides a set of natural 
coordinates for every point on the square specifying which 
side the point is on, and where on that side it is located. In 


this sense, discovering the underlying wreath process of a 
shape can be understood as finding a meaningful coordinate 
system for describing parts of that shape. This principle 
can be generalized to other structures, including finite state 
automata, and the stochastic wreath process and associated 
inference might find applications in such other domains, for 
example in the analysis of genetic regulatory networks as 
outlined in m 
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